Myth and Reality: Usage Behavior in a Large Data-Intensive Physics Project

1 Motivation Sustained effort is ongoing to support various scientific communities and their large-scale data-sharing and data-analysis needs through a distributed, transparent infrastructure (GriPhyN project [2], PPDG [13], EU DataGrid [5], etc). This infrastructure necessarily includes components for file location and management as well as for computation and data transfer scheduling. However, there is little information available on the specific usage patterns that emerge in these data-intensive, scientific projects. As a consequence, current designs either employ a “cover-all” approach (provisioning for all possible usage patterns and not optimizing for any), or start from unproved assumptions on user behavior by extrapolating results from other systems (such as file systems, the web, etc.). If inappropriate, these assumptions may lead to inadequate design decisions or to performance evaluations under unrealistic conditions.